Measuring success/failure in the installation

[Scott James Remnant]

On Wed, May 18, 2011 at 2:33 AM, Evan Dandrea <ev at ubuntu.com> wrote:

> To be clear, since it wasn't addressed in my original email, I intend
> to only present percentages of successful and unsuccessful installs.
>

As discussed externally, and now repeated here for the Technical
Board, this is the part of the proposal I have a problem with.

All raw data collected by this feature should be public.

There are three good reasons:

 1) Transparency.

    Making the raw data available makes it clear that the data
collected is anonymized and non-identifying. Users with concerns can
be shown the raw data URL and can verify for themselves that the data
does not identify them.

 2) Verification.

    With the raw data hidden, and only stats published by yourself,
there is no guarantee of honesty. If you claim that Ubuntu has 97%
successful installs, and somebody doubts that, they cannot go back to
the raw data and verify your results.

 3) Collaboration.

    This, to my mind, is the most important.

    Hiding the raw data, making it available only to yourself, makes
it harder for other developers to collaborate with you. Ubuntu is
still a community project.

    Say a developer wanted to not just look at installer success, but
on the average length of time in the installer, and undertake a
project to reduce it. With the raw data from this available, the
developer can trivially patch to add timestamps to the data set, and
analyse themselves for their project.

    With the data not available, that developer has to start from
scratch, including going though this procedure again.

Scott